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Why GAO Did This Study 

The Department of Homeland 
Security (DHS) has relied on 
service acquisitions to meet its 
expansive mission. In fiscal year 
2006, DHS spent $12.7 billion to 
procure services. To improve 
service acquisition outcomes, 
federal procurement law 
establishes a preference for a 
performance-based approach, 
which focuses on developing 
measurable outcomes rather than 
prescribing how contractors should 
perform services. 

GAO was asked to (1) evaluate the 
implementation of a performance- 
based approach in the context of 
service acquisitions for major, 
complex investments, and (2) 
identify management challenges 
that may affect DHS's successful 
acquisitions for major investments, 
including those using a 
performance-based approach. 

GAO reviewed judgmentally 
selected contracts for eight major 
investments at three DHS 
components totaling $1.53 billion in 
fiscal years 2005 and 2006; prior 
GAO and DHS Inspector General 
reviews; management documents 
and plans; and related data, 
including 138 additional contracts 
for basic services. 



What GAO Recommends 



DHS generally concurred with 
GAO's recommendations that DHS 
develop measurable standards 
consistently linked to well-defined 
requirements, evaluate acquisition 
outcomes for major investments, 
and improve data quality to help 
identify and assess contracting 
methods and outcomes. 

To view the full product, including the scope 
and methodology, click on GAO-08-263. 
For more information, contact John Hutton 
(202)512-4841 orhuttonj@gao.gov. 



What GAO Found 

All service contracts for the eight major, complex investments GAO reviewed had 
outcome-oriented requirements; however, four of these contracts did not have 
well-defined requirements, a complete set of measurable performance 
standards, or both. These service contracts experienced cost overruns, 
schedule delays, or did not otherwise meet performance expectations. In 
contrast, service contracts for the other four investments GAO reviewed had 
well-defined requirements linked to measurable performance standards. 
Contractors had begun work on three of these four contracts and performed within 
budget meeting the standards. This finding is consistent with prior GAO work 
on service acquisitions, which has highlighted the criticality of sound 
acquisition planning to develop well-defined requirements and measurable 
performance standards to achieving desired outcomes. In the four cases that 
had negative outcomes, program officials identified the contractor 
performance weaknesses through quality assurance surveillance and took 
corrective actions. Prior GAO work has found that if acquisitions, including 
those that are performance-based, are not appropriately planned, structured, 
and monitored, there is an increased risk that the government may receive 
products or services that are over budget, delivered late, and of unacceptable 
quality. 

In managing its service acquisitions, including those that are performance 
based, DHS has faced workforce and oversight challenges. Prior GAO work 
has highlighted the importance of having the right people with the right skills 
to achieve successful acquisition outcomes. Contracts for two major 
investments with negative cost and schedule outcomes did not have the staff 
needed to adequately plan and execute the contracts. Further, while 
representatives for several of the contracts GAO reviewed indicated that 
contracting and program staff worked well together, some senior acquisition 
representatives at the component level indicated that a lack of collaboration 
between these key stakeholders has been a challenge when developing and 
managing complex service acquisitions. In terms of oversight, component 
contracting and program officials said they used a performance-based approach to 
the maximum extent practicable; however, DHS does not have reliable data to 
facilitate required reporting, informed decisions, and analyzing acquisition 
outcomes. GAO's review also found that about half of an additional 138 
contracts for basic services identified as performance-based did not have any 
of the elements intended to foster good outcomes: a performance work 
statement, measurable performance standards, and a quality assurance 
surveillance plan. DHS's Chief Procurement Officer (CPO) — who is 
responsible for departmentwide oversight of acquisitions — has several efforts 
under way to address some of these workforce and oversight issues. One 
initiative is an acquisition oversight program that is intended to assess (1) 
compliance with federal acquisition guidance, (2) contract administration, and 
(3) business judgment. However, this oversight program has not yet included 
an evaluation of the outcomes of contracting methods such as performance- 
based service acquisition. 
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United States Government Accountability Office 
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April 22, 2008 

The Honorable Joseph I. Lieberman 
Chairman 

The Honorable Susan M. Collins 
Ranking Member 

Committee on Homeland Security and Governmental Affairs 
United States Senate 

The Honorable Daniel K. Akaka 
Chairman 

The Honorable George V. Voinovich 
Ranking Member 

Subcommittee on Oversight of Government Management, the 

Federal Workforce and the District of Columbia 
Committee on Homeland Security and Governmental Affairs 
United States Senate 

The Honorable Bennie G. Thompson 
Chairman 

Committee on Homeland Security 
House of Representatives 

In fiscal year 2006, the Department of Homeland Security (DHS) spent 
$12.7 billion on services, representing more than 80 percent of its total 
procurement spending for that year. DHS has relied on service acquisitions 
to meet its expansive homeland security mission. However, prior GAO 
work has found that if acquisitions are not appropriately planned, 
structured, and monitored, there is an increased risk that the services 
provided will not fulfill intended acquisition outcomes or, ultimately, meet 
government needs. 1 

To help improve service acquisition outcomes, federal procurement policy 
calls for agencies to use a performance-based approach to the maximum 
extent practicable. This approach includes: a performance work statement 
that describes outcome-oriented requirements, measurable performance 
standards, and quality assurance surveillance. These characteristics, if 
properly implemented, can help ensure that the services meet cost, 



1 GAO, Defense Acquisitions: Tailored Approach Needed to Improve Service Acquisition 
Outcomes, GAO-07-20 (Washington, D.C.: Nov. 9, 2006). 
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schedule, and performance requirements. Since this concept was 
introduced in the 1990s, it has become widely accepted as a sound 
acquisition approach. 

Despite its governmentwide acceptance, concerns have been raised about 
how well agencies are using a performance-based approach, particularly 
for complex or major investments. Given the significant amount of DHS 
spending on services and its critical mission, you asked us to (1) evaluate 
the implementation of a performance-based approach in the context of 
service acquisitions for major, complex investments; and (2) identify 
management challenges that may affect DHS's successful implementation 
of service acquisitions for major investments, including those using a 
performance-based approach. 

To conduct our work, we focused on the Coast Guard, Customs and 
Border Protection (CBP), Immigrations and Custom Enforcement (ICE), 
and the Transportation Security Administration (TSA) — four of the five 
DHS components reporting the highest obligations for performance-based 
service acquisitions in fiscal years 2005 and 2006. 2 To evaluate DHS's 
implementation of a performance-based approach in the context of service 
acquisitions for major investments, 3 we judgmentally selected eight major 
investments at the Coast Guard, CBP, and TSA 4 based on size and 
complexity, and reviewed actions for one performance-based service 
contract associated with each investment. 5 These contracts had a 
combined estimated value of $1.53 billion and included services provided 
in the acquisition and sustainment phases of the investments. 6 We assessed 
these contracts against acquisition regulations and policy and interviewed 
contracting and program management staff. Performance-based service 



2 We excluded the Federal Emergency Management Agency (FEMA) because of atypical 
fiscal year 2006 spending on Gulf Coast hurricane relief efforts. 

3 DHS has categorized major investments as Levels 1 and 2 and Level 3 (information 
technology). For the purposes of our review, we focused on Level 1 investments. A Level 1 
major investment is defined in DHS Management Directive 1400 as an investment with 
greater than $100 million in total acquisition costs including planning, or for information 
technology investments, a life-cycle cost greater than $200 million. 

4 ICE officials told us that none of their major investments used a performance-based 
approach. 

5 We reviewed either the contract or selected orders on the contract. 

6 DHS's investment review process includes three phases: pre-acquisition, acquisition, and 
sustainment. The acquisition phase includes systems development, and the sustainment 
phase includes operations and maintenance. 
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acquisitions were difficult to identify and were not clearly distinct from 
other service acquisitions because they did not always include all required 
elements. Therefore, we reviewed prior GAO reports and DHS Inspector 
General reports to provide a broader context on DHS's major service 
acquisitions. 

To identify management challenges that may affect DHS's successful 
implementation of service acquisitions for major investments, we analyzed 
management documents and plans; staffing data; and oversight 
mechanisms, including the Federal Procurement Data System-Next 
Generation (FPDS-NG), the governmentwide database for procurement 
spending. From FPDS-NG, we judgmentally selected 138 contracts for 
primarily basic services coded as performance-based and awarded in fiscal 
years 2005 or 2006 at Coast Guard, CBP, ICE, and TSA with obligations 
greater than $1 million dollars and asked the components to verify 
whether these contracts included the required performance-based 
elements. Finally, we interviewed contracting and program management 
staff and DHS Chief Procurement Office (CPO) representatives and met 
with representatives from the Office of Management and Budget's Office 
of Federal Procurement Policy (OFPP). For more information on our 
scope and methodology, see appendix I. 

We conducted this performance audit from January 2007 to April 2008 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 



RGSllltS in Brief ^ service contracts for the eight major, complex investments we 

reviewed had outcome-oriented requirements, but the requirements were 
not always well-defined. Four of the eight contracts we reviewed did not 
have well-defined requirements, or a complete set of measurable 
performance standards, or both at the time of contract award or start of 
work. These service contracts experienced cost overruns, schedule delays, 
or did not otherwise meet performance expectations. In contrast, service 
contracts for the other four investments had well-defined requirements 
linked to measurable performance standards. Contractors had begun work 
on three of these four contracts and performed within budget meeting the 
standards. This finding is consistent with our prior work on service 
contracting, which has highlighted the criticality of sound acquisition 
planning to develop stable, well-defined requirements and measurable 
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performance standards to achieving desired outcomes. Components 
conducted quality assurance surveillance for the contracts we reviewed. 
For the four contracts that had negative outcomes, surveillance helped 
DHS components to identify contractor performance weaknesses and 
corrective actions were taken. Prior GAO work has found that if 
acquisitions, including those that are performance-based, are not 
appropriately planned, structured, and monitored, there is an increased 
risk that the government may receive products or services that are over 
budget, delivered late, and of unacceptable quality. 

In managing its service acquisitions, including those that are performance- 
based, DHS has faced workforce and oversight challenges. Our prior work 
has highlighted the importance of having the right people with the right 
skills to achieve successful acquisition outcomes. Contracts for two of the 
eight major investments we reviewed that had negative cost or schedule 
outcomes did not have the staff needed to adequately plan and execute the 
contracts. Further, while representatives for several of the contracts we 
studied indicated contracting and program staff worked together 
effectively, at the component level, senior acquisition representatives 
indicated that a lack of collaboration between the program and 
contracting offices in general has been challenging. In terms of oversight, 
DHS component contracting and program staff told us they used a 
performance-based approach to the maximum extent practicable; 
however, the department does not have reliable data to facilitate reporting 
or perform management assessments of these acquisitions. Moreover, 
while the CPO has implemented a departmentwide acquisition oversight 
program, the CPO has not performed management assessments of 
performance-based service acquisitions. Our review of an additional 138 
contracts for primarily basic services found that about half of the 
contracts identified as performance-based in FPDS-NG did not have any of 
the three elements: a performance work statement, measurable 
performance standards, and a quality assurance surveillance plan, which 
are intended to foster good outcomes. Inaccurate data limit DHS's 
visibility over service acquisitions and the department's ability to make 
informed acquisition management decisions. DHS's CPO has several 
efforts under way to address departmentwide workforce and oversight 
challenges. One initiative is an acquisition oversight program that is 
intended to assess (1) compliance with federal acquisition guidance, (2) 
contract administration, and (3) business judgment. However, this 
oversight program has not yet included an evaluation of the outcomes of 
contracting methods such as performance-based service acquisition. 
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To increase DHS's ability to improve outcomes for its service acquisitions, 
including those that are performance-based, we are recommending that 
the Secretary of Homeland Security take several actions. These actions 
include improving acquisition planning for requirements for major, 
complex investments to ensure they are well-defined, and developing 
consistently measurable performance standards linked to those 
requirements; systematically evaluating the outcomes of major 
investments and relevant contracting methods at a departmentwide level; 
and improving the quality of FPDS-NG data so that DHS can more 
accurately identify and assess the use and outcomes of various contracting 
methods. In written comments on a draft of this report, DHS generally 
concurred with our recommendations and provided some information on 
efforts under way to improve acquisition management. The department's 
comments are reprinted in appendix II. OMB did not comment on the 
findings or conclusions of this report. DHS and OMB each provided 
technical comments, which we incorporated as appropriate and where 
supporting documentation was provided. 



Background Over the last decade, the use of federal service contracting has increased 

" and now accounts for over 60 percent of federal procurement dollars 

spent annually. Prior GAO work has found that if acquisitions are not 
appropriately planned, structured, and monitored, there is an increased 
risk that the services provided will not fulfill intended acquisition 
outcomes or, ultimately, meet agency needs. 

A performance-based approach, if successfully implemented, can help 
improve service acquisition outcomes. This concept — introduced during 
the 1990s — represented a shift in emphasis from specifying methods in 
which contracted work should be performed to specifying acquisition 
outcomes. In 2000, federal procurement law established a performance- 
based approach as the preferred acquisition method for services. 7 The FAR 
requires all performance-based service acquisitions to include: 



7 Floyd D. Spence National Defense Authorization Act for Fiscal Year 2001, Pub. L. No. 106- 
398 § 821(a) (2000) required that the Federal Acquisition Regulation (FAR) be revised to 
establish a preference for the use of a performance-based approach in the acquisition of 
services, which was done in FAR 37.102(a), providing that performance-based acquisition 
is the preferred method for acquiring services and generally is to be used to the maximum 
extent practicable. 
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• a performance work statement that describes outcome-oriented 
requirements in terms of results required rather than the methods of 
performance of the work; 

• measurable performance standards describing how to measure 
contractor performance in terms of quality, timeliness, and quantity; 
and 

• the method of assessing contract performance against performance 
standards, commonly accomplished through the use of a quality 
assurance surveillance plan. 8 

In addition, federal procurement law established a preference for using 
firm fixed-price contracts or task orders — where a specified price is paid 
regardless of the contractor's incurred costs — when using a performance- 
based approach for service acquisitions. 9 A performance-based acquisition 
approach also calls for greater collaboration among procurement and 
program representatives throughout the contract period and often requires 
more time for acquisition planning and management than other 
contracting methods. Agencies may also involve the contractor in the 
multi-functional team responsible for developing performance-based 
acquisition planning documents to leverage the contractor's technical 
expertise. Including stakeholders with varied knowledge and skills enables 
acquisition teams to determine at the earliest point possible that all 
aspects of the acquisition are necessary, executable, and tailored to the 
transaction's risk level. Regardless of the contracting method, focusing on 
outcomes and collaboration among multiple stakeholders in the 
contracting process has been acknowledged as sound contract 
management. Other factors, such as pressure to get programs up and 
running, additional external requirements, and technological challenges 
also impact the ability to achieve good acquisition outcomes. 

A 1998 OFPP study on performance-based contracts — based largely on 
contracts for basic services, such as janitorial or maintenance services — 
showed that a number of anticipated benefits had been achieved: reduced 
acquisition costs, increased competition for contracts, and improved 



FAR 37.601; FAR 37.602(b); FAR 37.604. A fourth element, performance incentives, is 
required where appropriate. 

9 Floyd D. Spence National Defense Authorization Act for Fiscal Year 2001, Pub. L. No. 106- 
398, § 821(a) (2000); FAR 37.102(a). 
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contractor performance. 10 However, our prior work and the work of others 
has found that implementing a performance-based approach is often more 
difficult for complex acquisitions because agencies begin with 
requirements that are less stable, making it difficult to establish 
measurable outcomes. OFPP has noted in policy that certain types of 
services, such as research and development, may not lend themselves to 
outcome-oriented requirements. 11 CPO representatives also have noted 
that defining outcome-oriented requirements and measurable performance 
standards may be more challenging for certain types of services, such as 
research and development or professional and management support 
services. Further, complex service contracts, such as those for information 
technology, may need to have requirements and performance standards 
continually refined throughout the life-cycle of the acquisition for a 
contractor to deliver a valuable service over an extended period of time. 

To encourage agencies to apply a performance-based approach to service 
acquisitions, the Office of Management and Budget established 
governmentwide performance targets: 20 percent of eligible service 
contract dollars for fiscal year 2002, 40 percent for fiscal year 2005 and 
fiscal year 2006, 45 percent for fiscal year 2007, and 50 percent for fiscal 
year 2008. 

In 2007, the congressionally mandated Acquisition Advisory Panel 12 
reported on the implementation of a performance-based approach and 
concluded that agencies were not clearly defining requirements, not 
identifying meaningful quality measures, not effectively managing the 
contract, and had limited data to make informed decisions. The panel 
made recommendations to improve application of the performance-based 
approach in three areas: goal setting and management, guidance for 
acquisition staff, and data quality. 13 For example, the panel recommended 



Office of Management and Budget, Office of Federal Procurement Policy, A Report on the 
Performance-Based Service Contracting Pilot Project, May 1998. 

"Office of Federal Procurement Policy Memorandum, "Increasing the Use of Performance- 
based Service Acquisitions," September 7, 2004. 

12 Services Acquisition Reform Act of 2003, Title XIV, National Defense Authorization Act 
for Fiscal Year 2004, Pub. L. No. 108-136, § 1423 (2003); National Defense Authorization Act 
for Fiscal Year 2006, Pub. L. No. 109-163, § 843 (2006) (extended the deadline for the 
Panel's report by 6 months). 

13 Report of the Acquisition Advisory Panel to the Office of Federal Procurement Policy and 
the United States Congress, January 2007. 
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that the Office of Management and Budget adjust the governmentwide 
target to reflect individual agency assessments and plans. 

In May 2007, OFPP issued a memo directing agencies, at a minimum, to 
meet targets established and report on them in their management plans. In 
response, DHS's CPO, who is responsible for creating departmentwide 
policies and processes for managing and overseeing the acquisition 
function, established a performance-based target of 25 percent for fiscal 
year 2007, increasing to 40 percent by fiscal year 2010, that was included in 
DHS's Performance-Based Management Plan. Meeting these targets is a 
responsibility shared between the CPO and the component heads, in 
accordance with DHS' system of dual accountability for managing the 
acquisition function. 14 



Definition of 
Requirements and 
Performance 
Standards Influenced 
Contract Outcomes 



Requirements definition and performance standards influenced outcomes 
for performance-based contracts associated with eight major, complex 
investments. While all eight contracts for the investments we reviewed had 
outcome-oriented requirements, the requirements were not always well- 
defined. 15 Further, contracts for half of the investments did not have a 
complete set of measurable performance standards. Major investments 
with contracts that did not have well-defined requirements or complete 
measurable performance standards at the time of contract award or start 
of work experienced either cost overruns, schedule delays, or did not 
otherwise meet performance expectations. Conversely, contracts with 
well-defined requirements linked to measurable performance standards 
delivered results within budget and provided quality service. This finding 
is consistent with our body of work on service contracting, which has 



Responsibility for the acquisition function at DHS is shared between the CPO and each 
DHS component head. Eight DHS components have internal procurement offices with a 
Head of Contracting Activity (HCA) who reports directly to the component head and is 
accountable to the CPO. The eight components are: Coast Guard, CBP, FEMA, Federal Law 
Enforcement Training Center, ICE, Office of Procurement Operations, Secret Service, and 
TSA. The HCA for each component has overall responsibility for the day-to-day 
management of the component's acquisition function. See GAO, Department of Homeland 
Security: Progress and Challenges in Implementing the Department's Acquisition 
Oversight Plan, GAO-07-900 (Washington, D.C.: June 2007). 

15 In using the performance-based approach, sound contracting practices dictate that 
required contract outcomes or requirements be well-defined, providing clear descriptions 
of results to be achieved. FAR 2. 101 specifically provides that a performance work 
statement for performance-based acquisitions describe the required results in clear, 
specific, and objective terms with measurable outcomes. 
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emphasized the importance of sound acquisition planning to develop well- 
defined requirements and measurable performance standards to achieve 
desired results. Table 1 summarizes our analysis of the requirements, 
performance standards, and outcomes for the eight performance-based 
contracts we reviewed. Component program staff conducted surveillance 
for these contracts, and in the four cases that had negative outcomes, 
identified less than expected performance and took action. 



Table 1 : Key Characteristics of Eight Performance-Based Service Contracts 


Major investment by 
component 


Service 


Well-defined 
requirements 9 


Measurable 
performance 
standards" 


Outcomes 


Coast Guard 


Response Boat 
Medium 


Research, analysis, and 
financial and information 
management 


• 


• 


Contractor submitted all required documentation 
on time; met project management quality 
standards; and maintained electronic archiving 
and restoration standards. 


Customs and Border Protection 


Automated 

Commercial 

Environment 


Trade systems software 
development (task order 
23) 


O 


® 


Costs increased by 40 percent ($21.1 million). 
More than a year behind schedule; unplanned 
software redesign. 


National Prime 
Integration 


Maintenance of 
equipment used at 
border crossings, 
airports, and seaports 


• 


® 


Costs increased by 53 percent ($24 million). 
Maintenance wait times were longer than 
planned. 


SBInef 


Project 28 border 
surveillance systems 
development and fielding 


o 


® 


DHS rejected initial acceptance of Project 28. 
The project was delayed 8 months with final 
acceptance in February 2008. DHS noted that 
the contractor met the requirements, but the 
project did not fully meet DHS's needs and the 
technology will not be replicated in future SBInef 
development. 


Transportation Security Administration 


Electronic Baggage 
Screening Program 


Maintenance for 
explosive trace detection 
machines 


• 


• 


Contractor exceeded the performance standard 
for machine downtime with a score 1 hour less 
than required and operated at cost through the 
second quarter of fiscal year 2007. 


Screening Partnership 
Program 


Passenger screening 
services at one airport 


• 


• 


Contractor exceeded most performance 
standards; for example: threat detection 
performance and false alarm rates exceeded the 



quality standards. Contractor had cost underrun 
of 2.2 percent ($677,000). 
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Major investment by 
component 


Service 


Well-defined 
requirements 9 


Measurable 
performance 
standards" 


Outcomes 


Secure Flight 


Maintaining database 
used to screen airline 
passenger data 


• 


® 


Initial contractor planning reports were 
inadequate; system experienced operational 
downtime; surveillance reports identified poor 
contractor performance. Contractor generally met 
time frames and delivered within budget. 


Transportation Worker 

Identification 

Credential 


Issuing identification 
credentials to maritime 
workers 


• 


• 


Outcomes not available at the time of our review. 



Legend: • contract met or mostly met the criteria; ® contract partially met the criteria; O contract did 
not meet the criteria 



Source: GAO analysis. 

"Well-defined requirements should provide clear descriptions of results to be achieved at the time of 
the award or start of work and primary requirements should not change substantially following 
contract award. 

"The set of measurable performance standards for a contract enables the government to assess all 
aspects of the contractor's work in terms of quality, timeliness, and quantity. The contract's 
performance standards are also linked to the requirements. 



Although all contracts we reviewed for the eight major, complex 
investments had outcome-oriented requirements, contracts for four 
investments — two for systems development and two for operations and 
maintenance — did not have stable and well-defined requirements, a full 
complement of measurable performance standards, or both. These four 
contracts experienced schedule delays, cost increases, or other negative 
outcomes. Prior GAO and DHS Inspector General work on other DHS 
major investments have similarly found that when requirements were not 
well-defined, performance did not meet expectations. Our work has found 
that performance-based acquisitions must be appropriately planned and 
structured to minimize the risk of the government receiving services that 
are over cost estimates, delivered late, and of unacceptable quality. 16 
Specifically, we have emphasized the importance of clearly defined 
requirements to achieving desired results and measurable performance 
standards to ensuring control and accountability. 17 



GAO, Coast Guard: Observations on the Fiscal Year 2008 Budget, Performance, 
Reorganization, and Related Challenges, GAO-07-489T (Washington D.C.: Apr. 18, 2007). 

17 GAO-07-20; and GAO, Coast Guard: Status of Efforts to Improve Deepwater Program 
Management and Address Operational Challenges, GAO-07-575T (Washington D.C.: Mar. 
8, 2007). 



Contracts That Lacked 
Stable, Weil-Defined 
Requirements, Complete 
Measurable Performance 
Standards, or Both 
Experienced Negative 
Outcomes 
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Systems Development Efforts Tw° CBP systems development contracts lacked both well-defined 

requirements and measurable performance standards prior to the start of 
work. The first — ACE Task Order 23 project, a trade software 
modernization effort under a letter contract 18 — was originally estimated to 
cost $52.7 million over a period of approximately 17 months. DHS 
contracting representatives told us that some technical requirements were 
not fully defined within the original scope of work, and some new 
software requirements were added after contract award. Because many 
requirements were not fully defined in advance of the contract's award, 
the program was unable to establish clear, measurable performance 
standards and valid cost or schedule baselines for assessing contractor 
performance. The need to redefine contract requirements for the ACE 
program contributed to significant schedule delays and cost increases. 19 
ACE Task Order 23 was completed at the end of October 2007 with a 
project cost increase of approximately $21.1 million, or 40 percent, over 
the original estimate. As of April 2008, DHS reported that some portions of 
the work were delayed to better define requirements, and the final portion 
of the software was projected to be completed in June 2009, about 26 
months later than planned. 

The second, CBP's Secure Border Initiative Network (SBlnet) Project 28 
contract, was intended to help secure a section of the United States- 
Mexico border using a surveillance system. Project 28 was planned to be 
SBlnet 's proof of concept, as well as the first increment of the fielded 
SBlnet system. CBP awarded the Project 28 contract before the overall 
SBlnet operational requirements and system specifications were finalized. 
Additional design requirements for equipment installation were added 
after contract award. DHS's Inspector General reported in February 
2007 — more than 3 months after the Project 28 contract was awarded — 
that CBP had not properly defined SBlnet's operational requirements and 
needed to do so quickly to avoid rework of the contractor's systems 
engineering. Several performance standards were not clearly defined to 
isolate the contractor's performance from that of CBP employees, making 
it difficult to determine whether any problems were due to the 
contractor's system design, CBP employees, or both. Other performance 
standards were difficult to measure because CBP did not plan a controlled 



A letter contract authorizes a contractor to begin work and incur costs before reaching a 
final agreement on contract terms and conditions, including price. 

19 GAO, Information Technology: Improvements for Acquisition of Customs Trade 
Processing System Continue, but Further Efforts Needed to Avoid More Cost and 
Schedule Shortfalls, GAO-08-46 (Washington, D.C.: Oct. 25, 2007). 
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operational test, which is a test conducted in a separate, instrumented test 
area that resembles the border environment. CBP planned to conduct 
operational testing, which is less rigorous, after the system was fully 
accepted. As a result, it was not clear how CBP intended to measure 
compliance with the Project 28 standard for probability of detecting 
persons attempting to illegally cross the border. Another standard stated 
that the acceptable level of contractor performance would be determined 
in the future. Project 28, planned as an 8-month, $20 million firm fixed- 
price effort, 20 did not meet the June 2007 delivery date, and in August 2007 
CBP notified the contractor that it did not approve delivery because the 
system failed the initial acceptance test. DHS conditionally accepted the 
system in December 2007 and fully accepted it in February 2008. DHS 
noted that the contractor met the requirements; however, Project 28 
resulted in a product that did not fully meet user needs and its design will 
not be used as a basis for future SBlnet development. 21 In addition, DHS 
officials have stated that much of the Project 28 system will be replaced by 
new equipment and software. 

Operations and Maintenance Performance-based contracts for two other major investments — CBP's 
Efforts National Prime Integration (NPI) contract and a TSA Secure Flight 

contract — included well-defined requirements but did not have a full 
complement of measurable performance standards. These operations and 
maintenance contracts had less than expected contractor performance or 
significant cost increases. 

CBP's NPI contract for maintenance of inspection, detection, and 
surveillance devices at border crossings, airports, and seaports included 
well-defined requirements for conducting equipment deployment, 
maintenance, and replacement as well as engineering and logistics 
support. For example, the requirements for equipment corrective 
maintenance specify the contractor shall conduct maintenance, including 
parts and labor, as a result of equipment or system failure. However, the 
contract did not incorporate performance standards for post-maintenance 
inspections and other standards needed to fully assess the quality of 
equipment maintenance performed by the contractor. CBP representatives 
acknowledged the need for additional measurable performance standards 



The contractor testified in October 2007 before subcommittees of the House Homeland 
Security Committee, that it spent approximately double the original contract amount. 
Because this is a firm fixed price contract, absent changes or delays created by the 
government, CBP is not responsible for these additional costs. 

21 GAO, Secure Border Initiative: Observations on the Importance of Applying Lessons 
Learned to Future Projects, GAO-08-508T (Washington D.C.: Feb. 27, 2008). 
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addressing quality of equipment maintenance to help ensure good 
contractor performance. While CBP does not plan to incorporate 
additional performance measures until September 2008, CBP revised the 
contract documentation in December 2007 to emphasize program 
management. Regarding contract outcomes, during the last year of the 
contract, costs were about $24 million, or 53 percent, higher than original 
estimates, in part because of increases in the quantity of equipment being 
supported. Customer wait times have been much longer than expected for 
equipment maintenance, and wait time standards have been revised to 
between 24 and 96 hours depending on the type of equipment. Average 
wait times from September 2006 to September 2007 were between 118 and 
106 hours, not meeting the revised standard. 

A TSA Secure Flight contract for operations and maintenance of the 
federal terrorist watchlist database, which is used to screen airline 
passenger data, also experienced less than expected outcomes. The 
contract we reviewed contained well-defined requirements and 
measurable performance standards for operation, maintenance, repair, 
and upgrade of the system. For example, the requirements specify that the 
contractor shall provide on-site technical and management support. 
Although the contract had some measurable performance standards, the 
contract did not have the full complement of standards needed to monitor 
performance, according to program representatives. For example, one 
standard required no operational system downtime and the contractor was 
not always able to meet this standard. After we reviewed the performance 
standards, program representatives subsequently incorporated some 
additional, improved standards to provide increased visibility over how 
well the contractor responds to system outages. These standards are 
designed to measure the time required to switch from the primary to the 
secondary system for planned and unplanned system outages. 
Additionally, according to program and contracting representatives, the 
contractor initially delivered inadequate plans for staffing and operations, 
and did not always meet the operational system downtime requirement, 
but was within budget and generally met established time frames. 

Prior Reviews Have Similar The inability to manage requirements and meet cost, schedule, and 

Findings performance objectives is not limited to the components and systems we 

reviewed. Several prior GAO and DHS Inspector General reviews of major 
DHS investments using a performance-based approach point to such 
shortcomings. For example: 
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• In June 2007, we reported that DHS's performance-based contract for a 
departmentwide financial management system, eMerge2, did not have 
clear or complete requirements. 22 Consequently, the program 
experienced schedule delays and less than acceptable contractor 
performance and was ultimately terminated after a $52 million 
investment. 

• In March 2007, we reported that the Coast Guard's performance-based 
contract for replacing or modernizing its fleet of vessels and aircraft, 
Deepwater, had requirements that were set at unrealistic levels and 
then were frequently changed. 23 This resulted in cost escalation, 
schedule delays, and reduced contractor accountability. The DHS 
Inspector General has also indicated that numerous opportunities exist 
for DHS to make better use of sound practices, such as well-defined 
requirements. The DHS Inspector General recently concluded that 
Deepwater had poorly defined requirements and inadequate oversight 
that contributed to ineffective or inefficient results and increased 
costs. 24 

• In February 2006, DHS's Inspector General reported that TSA's 
Information Technology Managed Services contract had unclear 
requirements that contributed to negative outcomes, including an 85 
percent increase to nearly $834 million in DHS's 3-year cost estimate 
and a failure to deliver expected information technology capabilities. 25 



Contracts with Weil- 
Defined Requirements and 
Measurable Performance 
Standards Experienced 
Positive Outcomes 



Contracts for four other major investments we reviewed — one at the Coast 
Guard and three at TSA — had well-defined requirements and measurable 
performance standards. These requirements did not significantly change in 
scope after contract award. Contracts for program management support 
for the Response Boat-Medium program, maintenance of the Electronic 
Baggage Screening Program, and passenger and baggage screening for the 



22 GAO, Homeland Security: Departmentwide Integrated Financial Management Systems 
Remain a Challenge, GAO-07-536 (Washington, D.C.: June 21, 2007). 

23 GAO-07-575T. 

24 Department of Homeland Security Inspector General, Major Management Challenges 
Facing the Department of Homeland Security, OIG-08-11 (January 2008). 

25 Department of Homeland Security Inspector General, Transportation Security 
Administration's Information Technology Managed Services Contract, OIG-06-23 
(February 2006). 
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Screening Partnership Program met the performance standards and were 
within budget. We were unable to evaluate outcomes for the fourth 
contract associated with the Transportation Worker Identification 
Credential (TWIC) program because key work under the contract was 
delayed. 

Program management support and related services we reviewed for the 
Coast Guard's Response Boat-Medium investment had well-defined 
requirements, and two of three performance standards were measurable. 
The contracted services included research and project support, financial 
and information management, and Web site support. The requirements for 
research and project support clearly stated the contractor was to provide 
project planning and execution support; documentation, database, and 
internet support; and administrative support. The performance standards 
were linked directly to the requirements. The two measurable standards 
were for timely submission of documents and electronic records keeping. 
The Coast Guard did not indicate how it would measure contractor 
performance for the third standard addressing research analysis and 
advisory services. Coast Guard representatives said the contractor had 
fulfilled the requirements within budget and on time, but did not document 
contractor performance. 

A maintenance contract for TSA's Electronic Baggage Screening Program 
also had well-defined program management and technical requirements. 
For example, the requirements for maintenance clearly stated the 
contractor should perform preventative and corrective maintenance on 
detection machines, providing specific definitions for each of these tasks. 
The contract also had measurable performance standards linked directly 
to the technical requirements. Specifically, program representatives used 
the 'mean downtime' performance standard to calculate the average 
number of hours a machine was unable to perform its mission during a 
16-hour day as a means of evaluating how quickly the contractor restored 
the detection equipment to operation. For fiscal years 2006 through 2007, 
the contractor met or exceeded the mean downtime performance 
standard, which has become more rigorous each year. Additionally, the 
contractor has stayed within the budget established in the firm fixed-price 
contract. The current maintenance contract improved TSA's ability to 
manage costs. Previously, DHS's Inspector General found TSA's initial 
electronic baggage screening maintenance contracts did not follow sound 
contracting practices. The Inspector General recommended TSA take 
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action to control costs by modifying the type of contract used. 26 TSA 
subsequently withdrew the initial maintenance contract and issued four 
new contracts, one of which we reviewed, for maintenance of detection 
machines at over 400 airports in the United States. 

Contracted security services at the San Francisco International Airport for 
the Screening Partnership Program also had well-defined requirements, 
and all measurable performance standards corresponded to contract 
requirements. This was an improvement from our prior reviews of TSA's 
Screening Partnership Program, which found that TSA had not finalized 
performance-based planning documentation, including developing 
measurable performance standards. 27 The contract we reviewed contained 
well-defined requirements for gate, checkpoint, and baggage screening; 
basic preventative and corrective maintenance; initial, recurrent, and 
remedial training; and recruiting and supervising screeners. For example, 
the requirements for gate, checkpoint, and baggage screening services 
clearly stated the contractor should use technology and staff to prevent 
prohibited items from entering sterile areas of the airport and should work 
to minimize customer complaints while addressing in a timely manner any 
complaints received. The performance standards assessed how often 
screeners could successfully detect test images of prohibited items in 
checked baggage; the percentage of audited records and inspected 
equipment, property, and materials that were well-kept, operational, and 
recorded on maintenance logs; and whether all new hires received the 
required training before assuming their screening responsibilities. In terms 
of expected outcomes, the contractor achieved a 2.2 percent cost 
underrun during the first 5 months of the contract and exceeded most 
requirements. 

While the worker enrollment services contract for TSA's TWIC program 
for a tamper-resistant biometric credential for maritime workers included 
well-defined requirements and measurable performance standards, we 
were unable to review the contract's outcomes as key work under the 



Department of Homeland Security Inspector General, Evaluation of TSA's Contract for 
the Installation and Maintenance of Explosive Detection Equipment at United States 
Airports, OIG-04-44 (September 2004). 

27 GAO, Aviation Security: Preliminary Observations on TSA's Progress to Allow Airports 
to Use Private Passenger and Baggage Screening Services, GAO-05-126 (Washington, D.C.: 
Nov. 19, 2004); and GAO, Aviation Security: Progress Made to Set Up Program Using 
Private-Sector Airport Screeners, but More Work Remains, GAO-06-166 (Washington, D.C.: 
Mar. 31, 2006). 
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contract was delayed. 28 Prior GAO work found that during the TWIC 
program's prototype phase, the program contract costs doubled due to 
expanded requirements after contract award. 29 The contract we reviewed 
contained well-defined requirements for developing enrollment centers; 
providing information technology maintenance services; and setting up 
help desk services for TSA employees, maritime workers, and other TWIC 
stakeholders. For example, the requirements for enrollment centers 
clearly stated the contractor was to provide maintenance services; 
infrastructure support such as work stations; and operations, 
management, and administrative support. Moreover, the contract's 
performance standards were measurable and linked to the requirements. 
For example, the program developed performance standards assessing the 
amount of time the contractor took to respond to and resolve each caller's 
issue and to enroll maritime workers in the program. By using measurable 
performance standards linked to the contract's requirements, TSA has 
improved its ability to identify and address performance issues and avoid 
negative outcomes. 



Quality Assurance 
Surveillance Identified 
Less Than Expected 
Performance 



Agencies are required to ensure that contractors are providing timely and 
quality services and mitigate contractor performance problems. The FAR 
indicates that surveillance plans provide for contract quality assurance by 
specifying contractor work requiring surveillance and the method of 
surveillance. 30 While not all of the contracts we reviewed had a quality 
assurance surveillance plan in place at contract award, in all cases DHS 
components monitored and assessed contractor performance against the 
contract performance standards. In the four cases where we found poor 
contract outcomes, program staff identified and documented less than 
expected performance and took action as shown in table 2. 



TSA awarded a contract in January 2007 to issue biometric credentials to maritime 
workers. TSA officials told us it took more time than originally planned to bring the TWIC 
prototype enrollment system into alignment with new government security standards. 
During the course of our review, the schedule to initiate work was delayed from March to 
October 2007. 

29 GAO, Transportation Security: DHS Should Address Key Challenges before 
Implementing the Transportation Worker Identification Credential Program, GAO-06-982 
(Washington, D.C.: Sept. 29, 2006). 

30 FAR 37.604 and 46.401. 
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Table 2: Less Than Expected Performance Identified through Surveillance and Action Taken 



Major investment by 
component 


Service 


Performance 


Action taken 


Customs and Border Protection 


Automated Commercial 
Environment Task Order 
23 


Developing trade systems 
software 


Costs increased by 40 percent 
($21 .1 million) and contractor did 
not achieve project milestones 
according to schedule 


Contractor did not earn full 
incentive fees 


National Prime 
Integration 


Maintaining equipment used 
at border crossings, airports, 
and seaports 


Longer than expected 
maintenance wait times 


Contractor did not earn full award 
fees 


SBInef Project 28 


Developing and fielding border 
surveillance systems 


System failed the initial 
performance test and project was 
delayed 8 months with final 
acceptance in February 2008 


Contractor did not receive 20 
percent of contract payment and 
DHS did not initially accept the 
system 


Transportation Security Administration 


Secure Flight 


Maintaining database used to 
screen airline passenger data 


Inadequacies identified in initial 
contractor planning reports 
provided to TSA; operational 
downtime; and other forms of less 
than expected performance 


Contractor did not earn full award 
fees and DHS worked with 
contractor to increase staffing 
levels 


Source: GAO analysis. 



In two of these cases — ACE and Secure Flight — program representatives 
identified less than expected performance through the use of a quality 
assurance surveillance plan. In contrast, the NPI program office told us 
they did not have a quality assurance surveillance plan for the first year of 
the contract, but instead relied on award fee plan criteria to assess 
contractor performance. The SBlnet program office monitored Project 28's 
execution through other means, such as program reviews, technical 
interchange meetings, status and issue reporting, and observation of 
contractor testing. Our prior work has found that if performance-based 
acquisitions are not appropriately planned and structured, including 
surveillance planning, there is an increased risk that the government may 
receive products or services that are over budget, delivered late, and of 
unacceptable quality. 31 



31 GAO-07-489T. 
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Insufficient workforce and limited oversight of acquisition outcomes has 
presented significant challenges for DHS in implementing its service 
acquisitions, including those that use a performance-based approach. 
Having adequate staff who collaborate across functions, and reliable data 
are among the key success factors that we have identified for achieving 
intended outcomes for all types of service acquisitions. 32 However, DHS 
continues to lack contracting and program staff with the expertise needed 
to adequately plan or monitor contractor execution of requirements, and 
some component acquisition representatives indicated a lack of 
collaboration between the program and contracting offices. In terms of 
oversight, inaccurate data has limited departmentwide visibility into DHS's 
service acquisitions, including those that are performance-based. 



Our prior work has highlighted the importance of having the right people 
with the right skills to achieve successful acquisition outcomes. 33 However, 
DHS has not fully defined the types of positions or numbers of staff for 
each position for its acquisition workforce. CPO representatives identified 
acquisition staff shortages as one of the primary obstacles to successful 
acquisitions, including those that are performance-based. CPO 
representatives have established departmentwide interim staffing goals 
addressing minimum and optimal staffing levels for contract specialists for 
each component. As of February 2008, CPO representatives reported that 
approximately 61 percent of the minimum required staff, and 38 percent of 
the optimal level of contract specialists, were in place. 

Insufficient contracting and program office staff was a challenge for two 
of the investments we reviewed — SBlnet and ACE. In these two cases, 
contracts were not on schedule or contractor performance was otherwise 
less than expected. In November 2006 — the month following SBlnet 
Project 28's contract award — the DHS Inspector General reported that 
DHS did not have the capacity needed to effectively plan, oversee, and 
execute the SBlnet investment; administer its contracts; and control costs 
and schedule. 34 For example, SBlnet had a shortage of engineers, logistical 
personnel, contracting officers, and cost analysts. In January 2007, DHS's 
Investment Review Board confirmed there were critical staffing shortages 



' "GAO, Framework for Assessing the Acquisition Function at Federal Agencies, 
GAO-05-218G (Washington, D.C.: Sept. 2005), and GAO-07-20. 

33 GAO-07-20. 

34 Department of Homeland Security Inspector General, Risk Management Advisory for the 
SBlnet Program Initiation, OIG-07-07 (November 2006). 



DHS Has Faced 
Workforce and 
Oversight Challenges 
in Managing Its 
Service Acquisitions 



Sufficient and Experienced 
Staff Are Needed to 
Ensure Successful Service 
Acquisitions 
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in the SBlnet investment, including Project 28. In February 2007, we 
reported that SBlnet representatives expressed concern about finding an 
adequate number of program office staff with the right expertise. We 
subsequently reported in February 2008 that the SBlnet program office 
staffing level had reached 305 positions out of a staffing goal of 470 by the 
end of fiscal year 2008. 35 CBP representatives expressed concerns that 
staffing shortfalls could affect the agency's capacity to provide adequate 
contractor oversight. ACE program officials also indicated they were 
significantly understaffed, which prevented them from fully defining the 
requirements for ACE Task Order 23. Officials told us the investment had 
less than 10 percent of the staff they needed at the time the requirements 
were being defined. 

In its fiscal year 2007 departmentwide performance-based service 
acquisition management plan, DHS also emphasized the importance of 
filling program and contracting vacancies with staff experienced in 
developing and implementing the required elements of performance-based 
acquisitions. Contracting and program management staff for TSA's 
Screening Partnership and Electronic Baggage Screening programs stated 
that their previous experience helped them to develop well-defined 
requirements and measurable performance standards. In these cases, 
contracts were on budget and contract performance met or exceeded the 
government's requirements. Representatives from Coast Guard, CBP, and 
TSA noted success in supplementing expertise through acquisition support 
offices, also known as centers of excellence, to improve acquisition 
planning. Component representatives told us this support could help 
acquisition staff improve the quality of contract documents, including 
performance work statements. These support offices can serve as a force 
multiplier, allowing their high-demand skill-sets to be used across 
contracts and helping to improve contracting outcomes across the 
component. 

According to CPO representatives, hiring knowledgeable acquisition staff 
has been difficult due to a limited pool of qualified acquisition 
professionals and intense competition between the government and the 
private sector for those professionals. 36 The CPO has the following ongoing 
departmentwide initiatives to recruit, train, and define its acquisition 
workforce that have the potential to better position DHS to address 
acquisition workforce challenges. These include: 



JO GAO-08-508T. 

36 We have ongoing work on DHS's acquisition workforce and plan to report on these issues 
in the final product for that engagement. 
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• centralize recruiting activities across the department; 

• institute an acquisition intern program; 

• employ direct hire authority 37 and rehire annuitants; 38 

• reemphasize training opportunities and establishing a centralized 
acquisition workforce training fund; and 

• conduct workforce planning. 

The need for improved collaboration among the acquisition workforce is 
also a recurrent theme in our work on acquisition management. We have 
noted that acquisition groups should have procedures in place that 
empower staff to collaborate when procuring goods and services and have 
controls and incentives to ensure collaboration occurs. 39 In particular, 
leading organizations generally employ a multifunctional approach 
including contracting, finance, legal, and other participants as needed to 
help develop cost-effective acquisition approaches and help ensure 
financial accountability. When requirements are dynamic, as was the case 
for some of the investments we studied, there is a greater need for more 
sophisticated collaboration — a point echoed by CBP and TSA 
representatives who emphasize the importance of early collaboration for 
complex acquisitions. OFPP guidance also emphasizes that performance- 
based service acquisition is a collective responsibility involving 
representatives from budget, technical, contracting, logistics, legal, and 
program offices. 

For the eight major investments we reviewed, program and contracting 
offices jointly approved the acquisition plans, and representatives for 
several of the contracts we studied indicated they worked together 
effectively. However, at the component level, senior acquisition staff at 



'Services Acquisition Reform Act of 2003, Div. A, Title XIV, National Defense Authorization 
Act for Fiscal Year 2004, Pub. L. No. 108-136, § 1413 (2003) allowed federal agencies to 
exercise direct hire authority in accordance with regulations issued by the Office of 
Personnel Management. This authority was extended to September 30, 2012, by the 
National Defense Authorization Act for Fiscal Year 2008, Pub. L. No. 110-181, Div. A, Title 
VIII, § 853 (2008). 

3S General Services Administration Modernization Act, Pub. L. No. 109-313, § 4 (2006) 
granted agency heads authority, in consultation with the OPM and OFPP, to approve 
reemployment of annuitants in fields related to acquisition management. 

39 GAO-05-218G. 



Page 21 



GAO-08-263 DHS Service Acquisitions 



TSA and CBP indicated that collaboration between contracting and 
program offices in general has been challenging. For example, they cited 
poor timing when bringing together key stakeholders in the contracting 
process, and in some cases, the program offices' preference for 
prescriptive requirements rather than a more outcome-oriented approach. 
Coast Guard and TSA acquisition representatives we spoke with indicated 
they have ongoing efforts to promote greater collaboration. 



Limited Data Inhibits 
DHS's Ability to Oversee 
Its Service Acquisitions 



While DHS contracting and program representatives told us they use a 
performance-based approach to the maximum extent practicable, DHS 
does not have reliable data, either from FPDS-NG or at a departmentwide 
level, to systematically monitor or evaluate service acquisitions, including 
those that are performance-based acquisitions. Reliable data are essential 
to overseeing and assessing the implementation of contracting 
approaches, acquisition outcomes, and making informed management 
decisions. 



Our review of 138 contracts at the Coast Guard, CBP, ICE, and TSA coded 
in FPDS-NG as performance-based illustrates the difficulty in identifying 
such contracts. 40 According to contracting representatives at the Coast 
Guard, CBP, ICE, and TSA, only 42, about 30 percent, of these contracts 
could be confirmed as containing the required performance-based 
elements — a performance work statement, measurable performance 
standards, and a method of assessing contractor performance against 
performance standards. About 18 percent had some but not all of the 
required performance-based acquisition elements, and about 51 percent — 
totaling about $347.3 million — had none of the required elements (see 
table 3). This analysis indicates that the FPDS-NG data are not reliable for 
reporting on the performance target for eligible service obligations. For 
example, in fiscal year 2006, DHS reported awarding performance-based 
contracts for 21 percent of eligible service contract dollars. However, due 
to unreliable data, DHS is likely farther away from meeting the 
governmentwide target than previously reported. In addition, in July 2006 
OFPP requested all agencies to report in their 5-year management plan on 
the use of performance-based contracts by service types — ranging from 
basic, such as janitorial and landscaping, to complex, such as information 
technology or systems development. The Acquisition Advisory Panel and 



40 The 138 contracts were awarded in fiscal years 2005 or 2006 with obligations greater than 
$1 million, totaling about $1.3 billion. 
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DHS's CPO also have raised concerns regarding the accuracy of the 
performance-based designation in FPDS-NG. The Acquisition Advisory 
Panel reported from its review at 10 federal agencies that 42 percent of the 
performance-based contracts it reviewed had been incorrectly coded. 41 



Table 3: Review of Performance-Based Elements on Selected Contracts 


Performance-based 
elements 


Coast 
Guard 


Customs and 
Border 
Protection 


Immigration and 
Customs 
Enforcement 


Transportation 
Security 
Administration 


Total 
contracts 


Percent of 
total contracts 


All elements 


18 


3 





21 


42 


30.4 


Some elements 


16 





5 


4 


25 


18.1 


No elements 


20 


5 


34 


12 


71 


51.5 


Total 


54 


8 


39 


37 


138 


100 



Source: GAO analysis of DHS review of 138 contracts coded as performance-based in FPDS-NG. 



According to DHS contracting representatives, contracts were miscoded 
as performance-based for several reasons. Contracting staff may have 
coded contracts as performance-based without the presence of the 
required elements. Contracting staff also may not have followed data 
validation procedures. Finally, there are technical FPDS-NG issues. For 
example, DHS contracting and program staff said FPDS-NG automatically 
applies the same designation to all orders issued under the master 
contract although some individual orders may or may not be performance- 
based. 42 While DHS officials acknowledged this can occur, they stated that 
the data field in FPDS-NG could still be edited for orders issued under the 
master contract and that it was incumbent on DHS contracting 
representatives to review the designation for every contract action to 
ensure they are coded correctly. 

Inaccurate federal procurement data is a long-standing governmentwide 
concern. Our prior work and the work of the General Services 
Administration's Inspector General have noted issues with the accuracy 



Report of the Acquisition Advisory Panel to the Office of Federal Procurement Policy and 
the United States Congress, January 2007. 

42 Known as Indefinite Delivery Indefinite Quantity (IDIQ) contracts, these contracts 
provide for an indefinite quantity of supplies or services within stated limits during a fixed 
period of time. The government places orders for individual requirements. 
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and completeness of FPDS and FPDS-NG data. 43 The Office of 
Management and Budget has stressed the importance of submitting timely 
and accurate procurement data to FPDS-NG and issued memos on this 
topic in August 2004 and March 2007. 

Accurate FPDS-NG data could facilitate the CPO's departmentwide 
oversight of service acquisitions, including those that are performance- 
based. For example, the selection of a contract type is the principal means 
agencies have for allocating financial risk between the government and the 
contractor. For performance-based services, federal law establishes a 
preference for using firm fixed-price as opposed to cost reimbursable or 
time and materials contracts. 44 In addition, the CPO has instructed 
components to avoid using time-and-material contracts for services. We 
reviewed the contract type for the 42 contracts DHS confirmed were 
performance-based. Our analysis showed that DHS awarded about 70 
percent of these 42 contracts as fixed-price; these actions represented 
about 42 percent of total obligations for these awards. About 37 percent of 
total obligations were awarded as cost-reimbursable, and 7 percent were 
awarded as time and materials contracts (see fig. 1). 



GAO, Reliability of Federal Procurement Data, GAO-04-295R (Washington, D.C.: Dec. 30, 
2003); GAO, Improvements Needed to the Federal Procurement Data System-Next 
Generation, GAO-05-960R (Washington, D.C.: Sept. 27, 2005); and General Services 
Administration Inspector General, Review of the Federal Procurement Data System-Next 
Generation (FPDS-NG), Report Number A040127/O/T/F06016 (March 2006). 

44 For fixed price contracts, a specified price is paid regardless of the contractor's costs, 
minimizing the financial risk to the government. For cost reimbursable and time and 
materials contacts, the government generally assumes the risk of cost overruns. For 
example, cost reimbursable contracts provide for the government to pay reasonable, 
allowable and allocable costs incurred by the contractor up to the contract's price ceiling. 
Time and materials contracts provide for acquiring services on the basis of fixed costs for 
labor hours and materials at the stated contract ceiling price. In time and materials 
contracts, if services delivered do not meet contract requirements and the government 
exercises its right to have the contractor correct the deficiencies, the government pays the 
additional labor and material costs to do so, excluding profit. 
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Figure 1 : Percent of Contracts and Obligations by Contract Type for 42 DHS 
Performance-Based Service Acquisitions (Fiscal Years 2005 and 2006) 
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Source: GAO analysis of DHS confirmed performance-based service acquisitions. 

""Other" includes contracts for which the contract type field in FDPS-NG was blank, designated as 
being none of the contract types listed within FPDS-NG, or coded with more than one contract type. 



At a departmentwide level, CPO representatives responsible for 
acquisition oversight indicated they have not conducted systematic 
assessments including costs, benefits, and other outcomes of a 
performance-based approach. To improve the implementation of 
performance-based acquisitions, CPO representatives established a work 
group in May 2006 to leverage knowledge among DHS components. They 
also noted that they are working with OFPP to develop a best practices 
guide on measurable performance standards and to gather good examples 
of performance-based contracts. In addition, the CPO has implemented a 
departmentwide acquisition oversight program that is intended to assess 
(1) compliance with federal acquisition guidance, (2) contract 
administration, and (3) business judgment. This program was designed 
with the flexibility to address specific procurement issues, such as 
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performance-based service acquisitions, and is based on a series of 
component-level reviews. 45 However, this oversight program has not yet 
included an evaluation of the outcomes of this acquisition method. 



As part of its expansive homeland security mission, DHS spends billions of 
dollars on service acquisitions for critical trade, transportation, and border 
security investments. Consistent with federal procurement policy, DHS 
has emphasized a performance-based approach to improve service 
acquisition outcomes. However, in keeping with our prior findings, DHS's 
designation of a service acquisition as performance-based was not as 
relevant as the underlying contract conditions. Sound acquisition 
practices, such as clearly defining requirements; establishing 
complementary measurable performance standards; and planning and 
conducting surveillance in order to take corrective actions are all 
hallmarks of successful service acquisitions. In the cases we reviewed, as 
well as in prior findings lacking these key elements, DHS did not always 
achieve successful acquisition outcomes. To improve DHS's acquisition 
management, the CPO is taking steps to address departmentwide 
workforce and oversight needs. However, improved data accuracy and 
systematic evaluation of contracting methods and outcomes are also key 
to successful oversight efforts. Continued emphasis on these areas is 
required to prevent less than expected acquisition outcomes for the 
department's critical mission. 



Recommendations for ^° mcrease DHS's ability to achieve improved outcomes for its service 

acquisitions, including those that are performance-based, we recommend 

Executive Action that the Secretary of Homeland Security implement the following three 

actions: 

• routinely assess requirements for major, complex investments to 
ensure they are well-defined and develop consistently measurable 
standards linked to those requirements; 

• at a departmentwide level, systematically evaluate the outcomes of 
major investments and relevant contracting methods; and 
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• continuously improve the quality of FPDS-NG data to facilitate the 
ability to accurately identify and assess the use and outcomes of 
various contracting methods. 



Agency Comments 
and Our Evaluation 



We provided a draft of this report to DHS and OMB for review and 
comment. In written comments, DHS generally concurred with our 
findings and recommendations and provided some information on efforts 
under way to improve acquisition management. The department's 
comments are reprinted in appendix II. OMB did not comment on the 
findings or conclusions of this report. DHS and OMB each provided 
technical comments, which we incorporated as appropriate and where 
supporting documentation was provided. 



In response to our recommendations that DHS (1) routinely assess 
requirements for major, complex investments to ensure they are well- 
defined, and develop consistently measurable standards linked to those 
requirements, and (2) systematically evaluate the outcomes of major 
investments and relevant contracting methods, DHS stated that the CPO is 
working to strengthen acquisition and procurement. For example, DHS 
noted that the CPO is revising the investment review process; reviewing 
major programs and investments; and building the capability to manage 
complex efforts by ensuring program offices are properly structured and 
staffed with the right people and skills. To assist in accomplishing these 
goals, DHS noted that the CPO has established a new division to provide 
oversight and support for acquisition programs. We have ongoing work at 
DHS reviewing the results of DHS's oversight of major acquisitions and 
plan to report on these initiatives in the final product for that engagement. 
Improving acquisition management has been an ongoing challenge since 
the department was established and requires sustained management 
attention. However, DHS's response did not address how the CPO's 
process and organizational changes will impact component-level 
management and assessment of complex acquisitions to improve 
outcomes. 



Concerning the third recommendation, that DHS continuously improve the 
quality of FPDS-NG data to facilitate the ability to accurately identify and 
assess the use and outcomes of various contracting methods, DHS stated 
that as part of the CPO's oversight reviews, the accuracy of the FPDS-NG 
data is validated for the review sample, including whether contracts have 
been properly coded as performance-based. DHS also added that the CPO 
is an active member of the OFPP group working to improve FPDS-NG and 
has established a governance board whereby the CPO reaches out to DHS 
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components to improve data collection. While these initiatives may be 
steps in the right direction, DHS's response did not present the results of 
the oversight reviews, or state how these coordination efforts address the 
causes of miscoding or how they will improve the quality of FPDS-NG data 
in the future. 

DHS also noted other initiatives to improve the acquisition workforce, 
including working to obtain qualified acquisition professionals and 
conducting staffing studies to better define the department's needs. We 
also have ongoing work at DHS on the acquisition workforce and plan to 
report on the results of DHS's efforts in the final product for that 
engagement. 



As agreed with your offices, unless you publicly announce the contents of 
this report, we plan no further distribution for 30 days from the report 
date. At that time, we will send copies of this report to the Secretary of 
Homeland Security, the Director of the Office of Management and Budget, 
and other interested congressional committees. We will also make copies 
available to others upon request. In addition, this report will also be 
available at no charge on GAO's Web site at http://www.gao.gov. 

If you have questions about this report or need additional information, 
please contact me at (202) 512-4841 or huttonj@gao.gov. Contact points 
for our Offices of Congressional Relations and Public Affairs may be found 
on the last page of this report. 




John P. Hutton, Director 

Acquisition and Sourcing Management 
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Appendix I: Scope and Methodology 



To evaluate selected Department of Homeland Security (DHS) 
components' implementation of a performance-based approach in the 
context of service acquisitions for major investments, we conducted a 
detailed file review of eight judgmentally selected performance-based 
contracts for major, complex investments with a total value greater than 
$100 million. 1 To identify the components with the highest obligations 
using a performance-based approach in fiscal years 2005 and 2006, we 
used the Federal Procurement Data System-Next Generation (FPDS-NG) 
and selected four components — Coast Guard, Customs and Border 
Protection (CBP), Immigration and Customs Enforcement (ICE), and the 
Transportation Security Administration (TSA). 2 We excluded the Federal 
Emergency Management Agency (FEMA) from our review because of 
atypical fiscal year 2006 spending on Gulf Coast hurricane relief efforts. 

To identify contracts for major investments, we utilized several different 
approaches. 3 Three issues required us to modify our plans — the lack of 
information in FPDS-NG, inaccurate DHS contracting data on major 
investments, and inaccurate information provided by one DHS component. 
Because the FPDS-NG major program field was typically blank, we were 
unable to use this field to identify contracts associated with DHS major 
investments. As a result, we asked DHS to provide us with a list of 
performance-based contracts associated with the four components' major 
investments. DHS provided a list including 76 contracts for 16 major 
investments. Due to data inaccuracies related to the contract numbers on 
DHS's list, we were only able to identify performance-based contracts 
associated with two major investments. We selected one contract from 
CBP's Automated Commercial Environment and one from TSA's Secure 
Flight that had the highest obligations. To obtain additional contracts to 
review, we met with CBP and TSA contracting representatives and they 
identified five other major investments with performance-based contracts, 
and we included one from each investment in our review. Coast Guard 



^HS has categorized major investments as Levels 1 and 2 and Level 3 (information 
technology). For the purposes of our review, we focused on Level 1 investments. A Level 1 
major investment is defined in DHS Management Directive 1400 as an investment with 
greater than $100 million in total acquisition costs including planning, or for information 
technology investments, a life-cycle cost greater than $200 million. 

2 During our review TSA was exempt from the FAR and followed the Acquisition 
Management System, developed by the Federal Aviation Administration, which included 
acquisition policy providing that service contracts should incorporate performance-based 
contracting methods. 

3 We selected eight major investments and reviewed either a performance-based contract or 
selected orders on a performance-based contract associated with each investment. 
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Appendix I: Scope and Methodology 



contracting representatives told us none of their major investments used 
performance-based contracts. However, we randomly selected and 
reviewed four contracts and seven orders at the Coast Guard from DHS's 
list of major investments, and found that five of them were performance- 
based. Of these, we selected three performance-based orders from one 
contract for one major Coast Guard investment — the Response Boat- 
Medium — to include in our review. Lastly, ICE contracting representatives 
also reported that none of their major investments used a performance- 
based approach and provided us with their best examples of a 
performance-based contract. We reviewed these examples and found that 
none of them were associated with a major investment. As a result, we 
excluded ICE from our contract review. 

For the selected contracts, with a combined estimated value of $1.53 
billion, we reviewed documentation, including acquisition plans, 
performance work statements and statements of objectives, quality 
assurance surveillance plans, and government or contractor performance 
reports where available. Performance-based service acquisitions were 
difficult to identify and were not clearly distinct from other service 
acquisitions because they did not always include all required elements. 
Therefore, we reviewed prior GAO and DHS Inspector General reports to 
provide a broader context for service acquisition and other DHS major 
investments. We reviewed public laws, federal and agency acquisition 
regulations, and the Office of Management and Budget's Office of Federal 
Procurement Policy (OFPP) memoranda and best practices. We also 
interviewed procurement and program representatives to determine the 
extent to which the performance-based planning process influenced 
expected outcomes. 

To identify management challenges that may affect DHS's successful 
implementation of service acquisitions for major investments, including those 
that use a performance-based approach, we analyzed management 
documents and plans; staffing data; and oversight mechanisms, including 
the Federal Procurement Data System-Next Generation (FPDS-NG), the 
governmentwide database for procurement spending. From FPDS-NG, we 
judgmentally selected 138 contracts for primarily basic services coded as 
performance-based and awarded in fiscal years 2005 or 2006 at Coast 
Guard, CBP, ICE, and TSA with obligations greater than $1 million dollars 
and asked the components to verify whether these contracts included the 
required performance-based elements outlined in FAR subpart 37.6. Finally, 
we interviewed contracting and program management staff and DHS Chief 
Procurement Office (CPO) representatives and met with representatives 
from the Office of Management and Budget's Office of Federal 
Procurement Policy (OFPP). To evaluate the extent to which DHS used 



Page 30 



GAO-08-263 DHS Service Acquisitions 



Appendix I: Scope and Methodology 



different contract types for performance-based acquisitions, we reviewed the 
contract type on all contracts DHS verified as performance-based. 

We conducted this performance audit from January 2007 to April 2008 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 
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Appendix II: Comments from the Department 
of Homeland Security 



US Department of Homeland Security 

Washington, DC 20528 




Homeland 
f Security 



April 11,2008 



Mr. John P. Hutton 
Director 

Acquisition and Sourcing Management 
U.S. Government Accountability Office 
441 G Street, NW 
Washington, DC 20548 

Dear Mr. Hutton: 

Thank you for the opportunity to review and comment on the Government Accountability 
Office's (GAO's) draft report GAO-08-263 entitled DEPARTMENT OF HOMELAND 
SECURITY: Better Planning and Assessment Needed to Improve Outcomes for Complex 
Service Acquisitions. Technical comments have been provided under separate cover. 

The Department of Homeland Security (DHS) generally concurs with the findings and 
recommendations in the draft report and offers the following with respect to the report's 
three recommendations: 

GAO Recommendation Ml: Routinely assess requirements for major, complex 
investments to ensure they are well-defined and develop consistently measurable 
standards linked to those requirements. 

GAO Recommendation #2: At a department-wide level, systematically evaluate the 
outcomes of major investments and relevant contracting methods. 

DHS Response to Recommendations #1 and #2: DHS is in the midst of many crucial 
acquisitions that are vital to its success. The Chief Procurement Officer (CPO) is 
working to strengthen acquisition and procurement by institutionalizing solid processes, 
including the following actions: 

A. Strengthening the requirements and investment review processes by improving the 
joint requirements council and Investment Review Board (IRB) process. We are 
preparing to initiate a new Department-wide requirements process and have 
reinvigorated our investment review process; 

B. Reviewing the major programs and investments to ensure that the requirements are 
clear, cost estimates are valid, technology risks are properly assessed, schedules are 
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of Homeland Security 



realistic, contract vehicles are proper, and the efforts are well managed. We have 
held one formal Deputy Secretary IRB and are projecting one per month. We are 
also beginning the processes to conduct paper IRBs and Deputy Undersecretary for 
Management IRBs, as well as establishing Acquisition Program Baselines (APBs) 
and authorizing execution to the APB for all Level 1 and 2 programs; 

C. Building the capability to manage complex efforts by ensuring that program offices 
are properly structured and staffed with the right people and skills to ensure 
efficient and effective program management and oversight; and to aggressively hire 
where we have known shortages; and 

D. Examining best practice metrics in use by other departments with the intent to start 
implementation this year. 

The Acquisition Program Management Division (APMD) of the Office of the 
Chief Procurement Officer (OCPO) began operations in August 2007. The division was 
established to provide oversight and support for acquisition programs. To date, APMD 
has performed Quick Look assessments of 37 Level 1 programs and has overseen Deep 
Dive reviews of the SBInet and Advance Spectroscopic Portal (ASP) programs. APMD 
has provided advice and guidance to a number of programs, particularly in the area of 
cost benefit analysis. Currently the APMD team is focused on an aggressive Investment 
& Acquisition process re-engineering effort. The effort includes replacing DHS 
Management Directive 1400 Investment Review Process, establishing revised investment 
and acquisition decision procedures, as well as processes for acquisition program 
baselining, periodic reporting, acquisition of services, and other initiatives as they are 
identified. 

CPO is also working to assure that DHS obtains qualified acquisition 
professionals at the right time with the right skill-set. Competition for these professionals 
is intense within the Washington, D.C. area. To resolve our personnel shortages, we are 
intensifying our human capital planning efforts to minimize skill and competency gaps as 
well as minimize our critical vacancies and reliance on contractors. For example, in 
response to the OMB 1102 Contracting Workforce Competency Gap Survey, we 
developed a training plan that spans the next three years. This training plan targets the 
contracting functional area within the DHS Acquisition Workforce, but it will also 
benefit other acquisition career fields including program management and Contracting 
Officer's Technical Representatives. 

CPO is also currently conducting staffing studies to better define our acquisition 
workforce needs. Currently our workforce includes program managers and contract 
specialists. As part of our human capital planning efforts, we will be identifying other 
required acquisition career fields such as test and evaluation, systems engineering, 
logistics, and cost estimating. We are aggressively working to ensure that each 
acquisition position, upon definition, is encumbered by an acquisition professional 
trained and certified at the appropriate level. To this end, we are continuously reviewing 
and updating our Acquisition Training Program, the underpinning of a good certification 
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program. We are utilizing the Defense Acquisition Workforce Improvement Act 
framework to develop DHS certification standards. We have also centralized a number 
of recruiting activities including issuing Department-wide vacancy announcements. Our 
centralized recruitment efforts to date have focused primarily on contracting 
professionals. Expansion to other acquisition career fields will occur as each series is 
defined and Department-wide needs are identified. This initiative supplements our 
Components' on-going recruitment efforts with a goal of recruiting the best candidates 
available. In 2005, the Department commenced the Acquisition Fellows Program. The 
goal of the Fellows Program was to attract new talent at the entry level into our 
acquisition positions, and retain and train them through a professional career 
development program. Building on the success of the Acquisition Fellows Program, CPO 
formalized and modeled it to further resemble the highly successful Department of 
Defense program. This year CPO received funding for the Acquisition Professional 
Career Program and 33 of the requested 66 positions were funded. In FY 2009, the plan 
provides for a total of 100 positions to be funded. Our inaugural class in the Acquisition 
Professional Career Program began in lanuary 2008 and a second class will begin in June 
2008. Our goal is to grow this program to 300 positions by FY 201 1 to fill our critical 
acquisition positions. 

GAO Recommendation #3: Continuously improve the quality of Federal Procurement 
Data System-Next Generation (FPDS-NG) data to facilitate the ability to accurately 
identify and assess the use and outcomes of various contracting methods. 

DHS Response to Recommendation #3: As part of CPO's oversight reviews, the 
accuracy of the FPDS data is validated for the review sample, including whether the 
contract has been properly coded as a performance based contract. In addition, OCPO is 
an active member of the Office of Federal Procurement Policy group that is working to 
improve FPDS. OCPO has also established a Governance Board whereby CPO reaches 
out to the DHS Components to improve data collection. 

Thank you again for the opportunity to comment on this draft report and we look forward 
to working with you on future homeland security issues. 




Sincerely, 



Penelope G. McCormack 
Acting Director 

Departmental Audit Liaison Office 
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